SSH remote Login

Linux basic operations

Visual Studio Code

Python

  • virtual environment

  • useful packages numpy, matplotlib, gvar, lsqfit

slurm and environment modules

SSH remote Login

SSH remote Login

Linux and MacOS

• In terminal
  ssh -p port user@address

  • port [optional] : port number
  • user: username
  • address: address of the server

  e.g ssh -p 10023 USR@222.200.137.16

Windows

• Powershell  
• Windows terminal

• Windows Subsystem for Linux (WSL)

Use ssh config file

• Open or create config file

  • Linux or MacOS
    /home/USR/.ssh/config

    USR: your username

  • Windows
    C:\USR\.ssh\config

• Append to config

  Host snsc
    HostName 222.200.137.16
    User USR
    Port 10023
  

• USR username

• ssh snsc

Two-Factor Authentication

• WeChat mini-program：数盾OTP

  Set your gesture

   

  Scan the QR code
  
  Generate the verification code

   

  Enter the verification code

   

  Enter the password

Linux Basic Operations

Linux Basic Operations

Path in linux

Operations

• Change directory cd DIR

  • DIR: absolute or relative

• List all contents in a directory ls DIR

  • DIR: absolute or relative

• List files with pattern

  e.g. ls DIR/*.py 🢩 list all files end with .py

  * match any

• Create an empty file: touch FILE

• Show file content: cat FILE

• Show first/last N lines of a file
  head -n N FILE  tail -n N FILE

• Capture content in file
  e.g. cat make.log | grep -C 3 error

  • -C: optional, show above and below 3 lines
  • | pipe

Operations

• 
  

  • Be Careful with rm -rf

  • One user account (USR) is shared by a group of participants

    You may inattentively delete files,dirs created by others

  • e.g. NEVER rm -rf ~/

    • ~/ is the shared home directory /public/home/USR

    • will remove other participants' files

  
  

• • Do not modify bashrc, bash_profile

    • Will affect other participants sharing the same user account

    • Use export ENV=VAL to set your environmental variables (usually not necessary)

Transfer files between remote and local

• scp [-r] SOURCE DESTINATION

  • -r (recursively) if SOURCE is a directory
  • e.g. scp -r ./demo snsc:~/
  • Do not support resume from break-point

• rsync [-r] SOURCE DESTINATION

  • add -v to show verbose info
  • e.g. rsync -rv ./demo snsc:~/
  • Support resume from break-point

• Compress and Uncompress

  • tar -czvf (c for compress)
  • tar -xzvf (x for extract)

Visual Studio Code  

Download and Install

• https://code.visualstudio.com/Download

Extensions

• Remote-SSH and Remote-Explorer
  or packed Remote-Development

• Language support extension, e.g.

  • Python, Python Debugger, vscode-numpy-viewer
• Indent Rainbow

  • colorize the indent in your code

  • python use indentation to identify function definition, loop structure, ...

Remote development using vscode

 • Use Remote-SSH extension

 • If warning popup, check Do not show agin and Allow

check Trust the authors ... folder home and Yes, I trust the authors

Enter verification code from 数盾OTP then password

Open remote directory using vscode

 

vscode command-line interface

• Open directory (default in new window)

  code DIR

• Open directory in current window

  code -r DIR

• see code --help

Python  

  

Numpy     matplotlib

 gvar   lsqfit

Python Virtual environment (venv)

• Create: python3 -m venv VENV

  • VENV: name of your venv

• Activate: source PATH_TO_VENV/bin/activate

• Deactivate: deactivate

Ready-to-go python venv

• The packages needed are already installed

• Activate the venv source /public/home/jiangxy/venv.public/bin/activate

• Deactivate: deactivate
  
  
  
  
  
  

Python REPL mode

• Read-Evaluate-Print-Loop

  source /public/home/jiangxy/venv.public/bin/activate

Python script mode

• python3 PATH_TO_SCRIPT ARGS

• e.g. python3 ./hello.py LPC

  import sys
  
  name = sys.argv[1]
  print("hello,", name)
  

• import: load packages

  like #include in c++

• argv[1]: 2nd arg after python3

  in this case: LPC

  • argv counter start from 0
  • argv[0] is ./hello.py

Working with vscode  

Set python interpreter in vscode

Numpy

Numpy

Numpy

• Broadcast mechanism

  • np automatically promotes the dimensions

       

Numpy

Export and Load npy file

data = np.random.rand(8,6,4)
np.save("./data.npy",data)

data_p = np.load("./data.npy")

print("sum(abs(diff)) =>", 
      np.sum(np.abs(data - data_p)))

• np.save and nup.load

extension: vscode-numpy-viewer

matplotlib

• An example

  import numpy as np
  import matplotlib.pyplot as plt
  
  x_ary = np.linspace(0,4,21)
  y_ary = np.exp(-x_ary)
  plt.plot(x_ary,y_ary,'--ob')
  #plt.plot(x_ary,y_ary,ls='--', marker='o', color='blue')
  plt.ylabel('exponential')
  plt.xlabel('t')
  
  plt.savefig("exp.png")
  

  

matplotlib

• axes and subplots

  import numpy as np
  import matplotlib.pyplot as plt
  t_ary = np.linspace(0,4,21)
  y_ary = np.exp(-t_ary)
  
  fig, (ax1,ax2) = plt.subplots(2,1,figsize=(4, 6)) 
  ax1.plot(t_ary,y_ary,'--or')
  ax2.semilogy(t_ary, y_ary)
  
  fig.savefig("subplots.png")
  

  

• fig, (ax1,ax2) = plt.subplots(2,1,figsize=(4, 6))

  • plt.subplots(2, 1 🢩 2 rows × 1 columns 🢩

matplotlib

• Errorbar plot

  import numpy as np
  import matplotlib.pyplot as plt
  
  t_ary = np.linspace(0,1.0,6)
  y_ary = np.array([1.,  0.82,0.67,0.54,0.45,0.37])
  y_err = np.array([0.06,0.06,0.04,0.05,0.05,0.04])
  plt.errorbar(t_ary,y_ary,y_err,ls='--', marker='o', 
               capsize=5, capthick=1, ecolor='red')
  plt.savefig("errbar.png")
  

  

• ls='--': line sstyle is dashed (--)

• marker='o': data point is marked by • ('o')

  try other styles, e.g. -, * ...

• capsize, capthick: , ecolor: color of errorbar

matplotlib anatomy

gvar (Gaussian Random Variables)

lsqfit (Nonlinear Least Squares Fitting)

$\chi^2$ definition

• Uncorrelated case $\dst \chi^2 = \sum_t \frac{\left[\bar{C}_t - f(t,\{p_i\})\right]^2}{\sigma_t^2}\hskip 4em$

  • $\bar{C}_t$ 🢩 mean value of the $t$-th data
  • $\sigma_t\,$ 🢩 standard deviation of the $t$-th data

  • No interference between different $t$ 🢪🢩 uncorrelated

• Correlated case $\dst \chi^2 = \sum_{t, t'}\left[\bar{C}_{t}-f(t,\{p_i\}) \right] \left(\mathrm{cov}^{-1}\right)_{t,t'} \left[\bar{C}_{t'}-f(t',\{p_i\})\right]$

  • $\dst \mathrm{cov}$ : covariance matrix

    $$\left(\mathrm{cov}\right)_{t,t'}=\frac{1}{N-1}\sum_{i}(C^{(i)}_t-\bar{C}_t)(C^{(i)}_{t'}-\bar{C}_{t'})$$

• See Prof. Qian Zhang's talk

lsqfit

• $\chi^2$ 🢩 measure the deviation between data and model

• least $\chi^2$ fit 🢩 optimize $\{p_i\}$ for minimal BUT resonable $\chi^2$

• lsqfit 🢩 a python package perform least $\chi$-square fit

  fit = lsqfit.nonlinear_fit(data=(var_val, dat_val), 
                             fcn=model, prior=ini_val, debug=False)
  

  • var_val: value of variables 🢩 $x$-axis
  • dat_val: value of datas (with erros) 🢩 $y$-axis (with err/cov)
  • model    : fit model function
  • ini_val : initial values of (also constrains on) fit parameters in model

lsqfit

An example

import gvar as gv
import lsqfit
import numpy as np
import matplotlib.pyplot as plt


data_ary = np.load("./data.npy") #2-d ndarray, shape 50x11
cntrl_ary = np.mean(data_ary,axis=0)
err_ary = np.std(data_ary,axis=0)
N_t = len(cntrl_ary)
t_ary = np.linspace(0,2,N_t)

t_ary_dct = {'demo': t_ary}
val_ary_dct = {'demo': gv.gvar(cntrl_ary, err_ary)}

def fit_model(t_dct, p):
    mdls = {}
    ts = t_dct['demo']
    mdls['demo'] = p['c0']*np.exp(-p['c1']*ts)
    return mdls

ini_prr = gv.gvar({'c0':'0(5)','c1':'0.2(0.5)'}) 

fit = lsqfit.nonlinear_fit(data=(t_ary_dct, val_ary_dct), \
                            fcn=fit_model, prior=ini_prr) 
print(fit.format(True))
print("\n=======\n fit.p => ", fit.p)

t_lst = np.linspace(0, 2, 50) 
fit_fcn_val = fit.fcn({'demo':t_lst}, fit.p)['demo']
fit_fcn_cntrl = gv.mean(fit_fcn_val)
fit_fcn_err = gv.sdev(fit_fcn_val)

fig, ax = plt.subplots(1,1, figsize=(10, 7))
ax.errorbar(t_ary, cntrl_ary, yerr=err_ary, fmt = 'ro', \
            alpha=0.4, label="$C_2$") 
ax.plot(t_lst, fit_fcn_cntrl, color="b", label="best fit") 
ax.fill_between(t_lst, fit_fcn_cntrl - fit_fcn_err, \
                fit_fcn_cntrl + fit_fcn_err,  alpha=0.3) 
plt.yscale("log")
plt.xlabel('t')
plt.ylabel('$C_2$')
ax.legend(loc='upper center', fontsize=10, frameon=True, \
            fancybox=True, framealpha=0.8, borderpad=0.3, \
            ncol=1, markerfirst=True, markerscale=1, \
            numpoints=1, handlelength=1.5)
fig.subplots_adjust(left=0.15, right=0.9, top=0.9, bottom=0.15)
fig.savefig("./lsqfit_demo.png")

Python dictionary

t_ary_dct   = {'demo': t_ary}
val_ary_dct = {'demo': gv.gvar(cntrl_ary, err_ary)}

def fit_model(t_dct, p):
  mdls = {}
  ts = t_dct['demo']
  mdls['demo'] = p['c0']*np.exp(-p['c1']*ts)
  return mdls

ini_prr = gv.gvar({'c0': '0(5)','c1': '0.2(0.5)'}) 

• dct = {key: value}, access the value bydct[key]

• Why dict ?

• Consider the case of

  • fit 2 datasets with 2 models
  • 2 models may have shared parameters

Combined fit

• Combined fit: e.g. 2 data sets, 2 models (with shared parameters)

  label	label1	label2
  data	$(\vec{x}^{(1)}, \vec{y}^{(1)})$	$(\vec{x}^{(2)}, \vec{y}^{(2)})$
  model	$f(x,\{\textcolor{blue}{\vec{p}},\vec{q}\})$	$g(x,\{\textcolor{blue}{\vec{p}},\vec{r}\})$
  ini_val	$\textcolor{blue}{\vec{p}_0}$, $\vec{q}_0$	$\textcolor{blue}{\vec{p}_0}, \vec{r}_0$

  • $\textcolor{blue}{\vec{p}}$ denotes the shared parameters

See Prof. Qian Zhang's talk

Result of fit

print(fit.format(True))

Least Squares Fit:
chi2/dof [dof] = 0.24 [11]    Q = 0.99    logGBF = 22.832

Parameters:
           c0   0.999 (14)     [    0 ± 5.0 ]  
           c1   0.996 (25)     [  0.20 (50) ]  *

Fit:
       key        y[key]      f(p)[key]
---------------------------------------
    demo 0    1.000 (18)     0.999 (14)  
         1    0.820 (19)    0.8184 (96)  
         2    0.669 (22)    0.6705 (73)  
         3    0.549 (17)    0.5494 (67)  
         4    0.450 (20)    0.4501 (69)  
         5    0.368 (17)    0.3688 (70)  
         6    0.304 (17)    0.3021 (71)  
         7    0.246 (20)    0.2475 (69)  
         8    0.204 (24)    0.2028 (66)  
         9    0.163 (17)    0.1662 (62)  
        10    0.135 (23)    0.1361 (58)  

Settings:
  svdcut/n = 1e-12/0    tol = (1e-08*,1e-10,1e-10)    (itns/time = 7/0.1s)
  fitter = scipy_least_squares    method = trf

fig.savefig("./lsqfit_demo.png")

What is a good-fit ?

• least $\chi^2$ fit 🢩 optimize $\{p_i\}$ for minimal BUT resonable $\chi^2$

• Resonable $\chi^2/\mathrm{d.o.f} \approx 1$

Least Square Fit:
chi2/dof [dof] = 13 [10]     
Q = 5.2e-24    
logGBF = -74.281

• Underfit
  Model failed to discribe data

Least Square Fit:
chi2/dof [dof] = 0.07 [10]   
Q = 1    
logGBF = -46.654

• Over fit
  Try extract too much info from data

Least Square Fit:
chi2/dof [dof] = 1 [10]      
Q = 0.41    
logGBF = -15.05

• Reasonable fit

Acess the Best fit

...
fit = lsqfit.nonlinear_fit(data=(t_ary_dct, val_ary_dct), \
                          fcn=fit_model, prior=ini_prr) 
t_lst = np.linspace(0, 2, 50) 
fit_fcn_val = fit.fcn({'demo':t_lst}, fit.p)['demo']
fit_fcn_cntrl = gv.mean(fit_fcn_val)
fit_fcn_err = gv.sdev(fit_fcn_val)
...

fit.p =>  {'c0': 0.999(14), 'c1': 0.996(25)}

• can we

try figure it out

Slurm Workload Manager  

Environment Modules  

Job submit: sbatch

• An example: ex1.sub.sh

  #!/bin/bash
  
  #SBATCH --job-name=Ex1.xnxiong   # job name
  #SBATCH --partition=cpueicc      # partition
  #SBATCH -n 1                     # totl cpu
  #SBATCH --ntasks-per-node=1      # cpu per node
  #SBATCH --output=%j.out
  #SBATCH --error=%j.err
  
  source /public/home/jiangxy/venv.public/bin/activate
  
  python3 ./Ex1.py > Ex1.log 2>&1
  

  > : redirect output stream
  2>&1: also append error message to Ex1.log

• Submit to Slurm: sbatch ex1.sub.sh

  Submitted batch job 908048

  • 908048: job ID

Manage your jobs

• Show job status: squeue --job JOB_ID

• Terminate a job: scancel JOB_ID

  • One user account is shared by a group of participants

    • You have the privilege to cancel others' job

  • Make sure only scancel your job

    • e.g. add your name as prefix to jobname in task.sh

      #SBATCH -J xnxiong_job
      

  
  

  Ref. http://iqm.scnu.edu.cn/a/20230308/278.html

• Different users may require different complier / library ...
• How to resolve conflict/incompatible :

• List loaded modules: module list

• List all available modules: module avail

• Modules needed during this summer school:
  
  

   module purge #unload all loaded modules
   module load gcc/10.3.0-gcc-4.8.5
   module load git/2.40.0-gcc-10.3.0
   module load cmake/3.22.2-gcc-10.3.0
   module load cuda/11.4.4-gcc-10.3.0
   module load hpcx/2.14/hpcx-mt
  

  put them in your job sumbmission script when needed

Exercise

1. Write a python script and submit to compute node

• Calculate $\dst y=e^{-t}$, for $t = 0.0, 0.2, 0.4, ...,2$
• Add Gaussian noise to each $\dst \{t_i,y_i\}$
  • noise 🢩 $0.02\cdot\xi$, where $\xi \sim N(0,1)$
  • how to sample from a given distribution, check numpy.random documentation
• Repeat Step 2 for 50 times, then you get 50 "samples"
• Export your "samples" to .npy file

2. Write another python script and submit to compute node

• Import previously generated .npy data file
• Calculate the mean value and standard deviation of each ${y_i}$
• Plot $\{t_i,y_i\}$ with error bars in linear-log plot
• Fit the data using model $\dst f(x,\{c_1,c_2\}) = c_1 e^{-c_2 t}$
• Plot the best fit and data together in one figure

3. Transfer the generated figure to your local machine

Tips: numpy, np.random, np.mean, np.std, gvar, lsqfit, matplotlib, scp/rsync

Python venv

source /public/home/jiangxy/venv.public/bin/activate

 module purge
 module load gcc/10.3.0-gcc-4.8.5
 module load git/2.40.0-gcc-10.3.0
 module load cmake/3.22.2-gcc-10.3.0
 module load cuda/11.4.4-gcc-10.3.0
 module load hpcx/2.14/hpcx-mt

#!/bin/bash

#SBATCH --job-name=Ex1.xnxiong   # job name
#SBATCH --partition=cpueicc      # partition
#SBATCH -n 1                     # totl cpu
#SBATCH --ntasks-per-node=1      # cpu per node
#SBATCH --output=%j.out
#SBATCH --error=%j.err

Documentaion